arXiv:1504.04092v3 [cs.IT] 7Jun2015 


One-Shot Mutual Covering Lemma and Marton’s 
Inner Bound with a Common Message 

Jingbo Liu Paul Cuff Sergio Verdii 

Dept, of Electrical Eng., Princeton University, NJ 08544 
{jingbo,cuff,verdu}@princeton.edu 


Abstract —By developing one-shot mutual covering lemmas, 
we derive a one-shot achievahility hound for broadcast with a 
common message which recovers Marton’s inner hound (with 
three auxiliary random variables) in the i.i.d. case. The encoder 
employed is deterministic. Relationship between the mutual 
covering lemma and a new type of channel resolvability problem 
is discussed. 

I. Introduction 

While many problems in network information theory have 
been successfully solved in the discrete memoryless case 
using either the method of types or weak/strong typicality 
(e.g., [1] or [2]), there is a gap between the i.i.d. assump¬ 
tions underlying these methods and the nature of sources 
and channels arising from real world applications. Recent 
works (e.g. [3] [4]) have developed new methods to derive 
tight one-shot achievahility bounds for specialized problems. 
Meanwhile, a natural question is whether there exist general 
techniques to attach non-asymptotic fundamental limits in 
multiuser information theory. By developing one-shot versions 
of covering and packing lemmas, [5] successfully obtained 
one-shot achievahility bounds for various problems (multi¬ 
access, Slepian-Wolf, Gelfand-Pinsker, Wyner-Ziv, Ahlsede- 
Korner, and broadcast of private messages) which lead to 
such non-asymptotic bounds and recover known results in the 
i.i.d. case. However the proof of Marton’s inner bound of 
broadcast without public/common messages (the two auxiliary 
version, [1, Theorem 8.3]) proceeded by showing the achiev- 
ability of each corner point, which requires time sharing to 
recover the full rate region in the i.i.d. case. 

In this paper we develop a one-shot mutual covering lemma 
so that a one-shot version of Marton’s inner bound with a 
common message (the three auxiliary version originally due 
to Liang et al. [6]; see also [1, Theorem 8.4]) can be obtained 
without time sharing. 

Time sharing may not be satisfactory since it is meaningless 
in the one-shot case. This is keenly noted by the authors of 
[7], who also observed that the mutual covering lemma [1, 
Lemma 8.1], a technique for avoiding time sharing in the 
i.i.d. case, does not seem to have a one-shot counterpart: 

... This is helpful because no one-shot extension of 
the mutual covering and packing lemma exists... For 
this reason Verdu’s result seems to be weaker than 
ours. 

The present paper gives a single-shot mutual covering lemma, 
thereby hlling the gap noted by the above remark. We will 


provide two derivations, one of them is based on a recent 
generalization of channel resolvability [8]. 

Partly motivated by the desirability of achievahility bounds 
without time sharing, [7] (see also [9]) proposed a new tech¬ 
nique for deriving one-shot achievahility bounds using stochas¬ 
tic encoders, thus avoiding the mutual covering lemma and 
yielding a one-shot bound which recovers the two auxiliary- 
variable version of Marton’s inner bound. However that ver¬ 
sion of Marton’s bound is known to not be tight, whereas 
the three auxiliary-variable version (Liang-Kramer bound) is 
still a candidate for the capacity region [1, Section 8.4]. In 
[9] a one-shot achievahility bound with three auxiliaries is 
stated without a proof', which is not easily comparable with 
our bound. However the main probability terms from the two 
bounds are the same, so they are equivalent in the second order 
rate analysis. 

IT One-shot Mutual Covering Lemma 

We develop one-shot mutual covering lemma(s) in this 
section, which require a different proof idea than the i.i.d. case 
[1, Appendix 8A] because the empirical distribution is mean¬ 
ingless in the non-block coding case. The main device in 
our proof is the introduction of an auxiliary random variable 
V, which can be viewed as being “typical” with a random 
codeword from the U codebook. 

To begin with, let us introduce the notations of the relative 
information 

dP 

*P||Q(a;) := log^(a:) (1) 

where P and Q are distributions on a same alphabet X 
with being the Radon-Nikodym derivative, and the 

information density 

iu-v{u] v) := {v). (2) 

Lemma 1. Fix Puv and let 

PpMvL := Pp X ■ ■ ■ X Pp X Py X ■ ■ ■ X Py (3) 

* The authors of [9] announced that the proof will be included in their future 
draft. 


then 


Thus 


M,L 


^ min{M,L}-l , „_expM 
+ ^ +e . 


P[ n {{Un.,Vi) i T}] 

< ¥[{11, V) ^ T] + P [exp(i£/;v(J7; F)) > MLexp(— 7 ) — 5] 

(4) 

for all 5 ,7 > 0 and event T. 

Proof: Let {Vi,..., Vm) be random variables satisfying 

Pjjmvm = Puv X • • • X Puv, (5) 

and define 

V := Vw ( 6 ) 

where W is equiprobable on {1,..., M} and independent of 
(U^, V^). By independence, 


M,L 


M 


n {(C/m, Vl) i P}] = EP[ fl {([/„, Fi) i P}\U^Y 




Define the functions 


m—l 


(7) 


M 


[1 ~ 


{L exp(- 7 ) ) exp( 7 ) 


n L 


< 


1 - 


1 - ^^^PME[7r([/^;y)|C/^] 




( 11 ) 

( 12 ) 

(13) 

(14) 


where 

• (13) is from the fact that (by change of measure) 

Lexp(— 7 )A[/m > Lexp(— 7 )E[ 7 r(( 7 '^; Li)|C7'“] (15) 
>¥.[k{U’^-V)\U^]. (16) 

• (14) uses the inequality 


(l-^)-<l-p + e 


(17) 


V.[1-Aum]^ (18) 

< E[1 - ^((7^; V)] + e- (19) 

M 

= — ^ E[1 - 7TiU^;Vw)\W = w] + (20) 

Li;=l 

= E[1-7r(C/^; Vl)]+e-®^P(T'l (21) 

< P[(C/i, Fl) ^ ^,or ZyM.p(C/'^; Fl) > logL - 7] + e-""P(^) 

( 22 ) 

<P[(F,F) 


+ . 


1 “ 

-F 

A/f ^ 


dP 


v\u=u„. 


M dPv 

m—l 


■(Fl) > Lexp(-7) 


_|_ g- exp(7) 

(23) 


< P[(F, V)iP]+] 


dP, 


M 

E 


dPv\U=Ur. 


dPv 


V\U=Ui 

dPv 

-(Fl) ><5 


Lm=2 

<P[(F,F) ^P] + 
M- 1 


dP 


V\U=Ui 

"dp^ 


_|_ g- exp(7) 


(Fl) > MLexp(— 7 ) — 5 
+ g-exp(7) (24) 

(Fl) > MLexp(— 7 ) — () 
(25) 


A„m :=P[U{(u„,Fi)€P}]; 

(8) 

• 

1 

1 

71=1 


• 

tt { u ^; v) := 1 < 

U {(u™,u) e P} \ 
lrM=l J 

(9) 

• 

• 1 < 

u) < log L - 7I. 

(10) 

• 


The second term in (10) will play a role in the change of 
measure step. We then obtain 


where 

( 21 ) is because by symmetry, all summands in ( 20 ) are 
equal. 

(23) is because by definition, P/j^y^ = Puv- 

(24) uses the elementary inequality l{x + y > a + b} < 
l{x > a} + \{y > h], 

(25) uses the Markov inequality. 

■ 

Remark 2. Supposing that M < L, an optimal choice of ( 7 , 6) 
is the solution to the optimization problem 

M -1 


minimize /( 7 , 6) := 


_|_ g- exp(7) 


(26) 


subject to (/( 7 , S) := MLexp(— 7 ) — S = X (27) 

for some constant A G K. Using Lagrange multipliers we can 
show that the optimal value of d for fixed 7 is: 

,5 = a/(M- l)MLexp(-7)e5®^p(T'). (28) 

Note that Lemma 1 recovers the covering lemma in [5] 
when min{M, L} = 1. Also, Lemma 1 can be weakened to 
the following simpler bound by setting 6 = ML(exp(— 7 ) — 
exp(- 27 )). 

Lemma 3. Under the same assumptions of Lemma 1, 

M,L 

P[ n mm,Vl)^P}] 

m—1,1=1 

< F[{U, F) ^ J-] + P [iu-y{U-, V) > log(ML) - 27 ] 


for M, a > 0 and 0 < p < 1 . 


min{M,L}-l , „-exp(7) 

/ \ / \\ “T ^ 


ML(exp(—7) — exp(—27)) 


(29) 



for all j > 0 and event T. 

Remark 4. In all versions of the mutual covering lemmas 
given above, the sum of the two probability terms can be 
strengthened to the probability of the union of two events, 
if the union bound is not applied to simplify ( 22 ). 

A conditional version of the mutual covering lemma 
(c.f. Lemma 8.1 in [1]) also follows by averaging: 

Lemma 5. Fix Pjjst and let 


Ptjs’^T^ Pu X Ps\u X • • • X Ps\u X Pt\u X • • • X Pt\U 

(30) 

then 

M,L 

P[ n {{U,Sm,Ti)iP}] 

m—1,/—1 

< P[{(C/, T)iP)\J {ts;T|t/(^; T\U) > log(ML) - 27 }] 

I min{M,L}-l exp(^) . 0 ,. 

ML(exp(—7) — exp(—27)) 

for all j > 0 and event T. 


III. Mutual Covering from Resolvability 

While the mutual covering lemma in Section II suffices the 
purpose of Section IV, in this section we provide a simple 
alternative derivation based on a recent result on resolvability 
in the excess information metric [ 8 ], which illustrates the 
interesting connection between resolvability and the mutual 
covering lemma^. 


Lemma 6. [8] Fix Puv = PuPv\u- c = [ci,..., cm] be 
i.i.d. according to Pu- Define 

1 ^ 

Pv-=mY.Pv\u=c^- ( 32 ) 

m—1 


Then for any 


< 


iv-,u{V;U) > log^ 


2 

A 


(33) 


where the expectation is with respect to the codebook realiza¬ 
tion, V Pv and {U, V) ~ Puv- 


The above lemma follows by setting y = \, a = e f 
t Qxu = Pvu^ ttx = Pv^ L = M and Px = Pv in [ 8 , 
Remark 3]. We then have 


Lemma 7. Under the same assumptions of Lemma 1, 


M,L 

p[ n ^ p}] 


< F[{U, V)iP]+ F[iu-,v{U] V) > log ML - 7] 

I exp(7) iexp(7)_ 

max{M, L} 


(34) 


^Indeed, [10] uses the term “soft-covering” for the achievability part of 
channel resolvability [11], an idea that traces back to Wyner [12]. 


Remark 8 . The bound in Lemma 7 appears similar to and 
slightly simpler than Lemma 3. However the sum of the 
two probabilities in (34) cannot be easily strengthened to the 
probability of a union (see Remark 4), which is important in 
the second order rate analysis. 

Remark 9. In both derivations of the one-shot mutual covering 
lemma, the role of U and V are asymmetric. Moreover, the 
two methods are not readily extendable to obtain a one-shot 
version of the multivariate covering lemma [13, Lemma 8.2]. 

Proof of Lemma 7: Assume without loss of generality 
that L > M. Define the sets 


Tu := {v : (m, v) £ P}, (35) 

M 

aIkM := [J Pu- (36) 

m=l 

For fixed c = c^ in Lemma 6 , 

PviAc) — XPviAc) < sup[A^(Al) — XPviA)] (37) 

A 

= P|^(l>)>A|-AP(^(V)>A| 

(38) 

<n^{V)>X] (39) 

where (38) is from Neyman-Pearson lemma. Thus 

XPviA.) > PviA.) - n^iV) > X] (40) 

1 ^ (\ P 

m—1 

(41) 


where (41) is from P.^ C A.- Denote by p. the right hand 
side of (41). Then p. < 1 , and setting c = we obtain 


nr\{iUm,Vi) i P}\U^] = [1 - Pv{AuM)t 



< 1 - Pc + e 


(42) 

(43) 


where (43) uses (17). Then the result follows by uncondition¬ 
ing on both sides of (43), applying (41) and Lemma 6 , 
and setting A = 2Lexp(— 7 ). ■ 

While the derivations of the one-shot mutual covering 
lemma in Sections II and III follow different routes, their 
correspondences can be seen in the following ways: 

1) The auxiliary random variable V, which is the main 

device in the first proof, has the distribution Py as in 
(32) conditioned on . 

2) The change of measure steps (16) can be related to the 
Neyman-Pearson lemma (37)-(39). 

3) Both derivations relies on the inequality (17) which also 
appeared in the proof of the standard covering lemma. 



Although Lemma 6 implies the one-shot mutual covering 
lemma, the reverse implication does not seem to follow 
directly. Thus resolvability in the excess information is a 
stronger result than the mutual covering lemma. 

IV. Inner Bound with a Common Message 

We prove a single shot version of the asymptotic achiev- 
ability result of Liang-Kramer [6, Theorem 5] (see also [1, 
Theorem 8.4]). This region is equivalent to an inner bound 
obtained by Gelfand and Pinsker [14] upon optimization (see 
[15] or [1, Remark 8.6]). 

Theorem 10. Fix arbitrary distributions Pust, o 

map X : U X S X P X, and integers Mq, Miq, M 20 , N, 
L, N and L. Set 

M := MqMiqM2o', (44) 

Ml := MioN; (45) 

M 2 := M20L; (46) 

N := NN] (47) 

L := LL. (48) 

Then there exists an (Mq, Mi, M2, ei, £2) code with 
max{ei, £2} 

< 2 exp(—7) + ( 49 ) 

+ P[{*(7S;Ui(C^<S';ri) < log MTV+ 7} 

U {tC/T;Y2 {UP ; ^2) < log ML -f 7} 
u {is-,Yi\u{S',Yi\U) < logiV-f 7} 
u {*T;F2|C/(r; V2IP) < logL + 7} 
U{ts;T|t/(5;T|C/)>log(iVL)-27}] 
minjTV, L| — 1 

+ - (50) 

TVL(exp(—7) — exp(—27)) 

where Pustxy^y.^ '■= PustPx\ustPyiY 2 \x- 

As in [1, Theorem 8.4], the private messages are decom¬ 
posed into a public part and an individual part: 

W, = {W,o,Wu) (51) 

where i = 1,2 and Wio is supposed to be decodable by both 
users. 

Proof: 

• Codebook Generation: generate 

u= [Ui,...,Um] (52) 

according to the distribution . Also for each 1 < 
i < M, generate 

Si = [Si(n, n)] (53) 

according to and 

ti = [ti((, 0]l<i<L,l<i<L (54) 


according to (In other words, for each i we 

construct a codebook similar to [1, Figure 8.8], where 
each small rectangle has size L x TV.) 

• Encoding: we may assume that the public message is 
and the private messages for the two users are (iwio, a) 
and {w 2 Q,b) respectively, where 


Wo € {1,. 

.,Mo} 

(55) 

Wio e {1,. 

• , Mio} 

(56) 

a € {1,. 

.,TV} 

(57) 

6 € {1,. 

.,L} 

(58) 


for i = 1,2. Then the index m = in(rt;o,tcio,W20) 
is selected for the lower-layer codebook where m is a 
bijection between [Mq] x [Mio] x [M20] and [Mj. The 
encoder then finds d € {1,..., TV} and b € {1,..., P} 
that minimize 

C(^m, S7n(o, d), tm(&, 6)) (59) 

where 

(}('U, S,f) ■— PYiY2lX—x(u,s,t)(Pu,s,t) (60) 

and 

yu,s,t := {(J/1,2/2) e yi X y2 ■■ t( 7 S;Ui(MS; 2 /l) < logMTV-l-7 
or tuT-,Y2{ut;y2) < log ML-f 7 
or ts-,Yi\u{s',yi\u) < log TV-f 7 
or ZT;Y2|u(i;2/2|u) < logL + 7} ( 61 ) 

The signal transmitted is then x{Um,Sm{a, d),im{b,b))- 

• Decoder: assume that yi and y 2 are observed by the two 
receivers, respectively. The decoder of the first receiver 
(Decoder 1) finds the unique m such that 

3d,d, ius-,Yi{^mSm{d,d)]yi) >\ogMN+ y (62) 

and set (mo,mio,m 2 o) = or declares an error 

if not possible. Decoder 1 then finds the unique (c, c) 
such that 

*S;Yi|c/(Sm(c,c);yi|u™) > log TV-f 7 (63) 

or declares an error if not possible. The output public 
message is then mo and the output private message is 
(mio,c). 

Decoder 2 performs similar operations as Decoder 1. 

• Error Analysis: By symmetry of the codebook, we may 
assume without loss of generality that 

Wo = wio = W 20 = tfii = W 22 = 1 (64) 

is sent and m(l, 1,1) = 1. Also it suffices to prove the 
bound for Decoder 1 only in view of the symmetry of 
the bound (50). Assume that (d, b) minimizes (59) (and 




is selected by the encoder). Decoder 1 fails only if one 
or more of the following events occur: 


ft : 

*c/S;Yi(UiSi(l,a);t/i) < logMiV + 7 ; 

(65) 

^2 : 

3m A 1, d, d, 




lUS-,Yi (UmSm 

,(d,d);t/i) > log MTV+ 7 ; 

( 66 ) 

^3 : 

*S;Yi|;7(Sl(l, 

d); 2 /i|Ui) < log TV+ 7 ; 

(67) 

£4 : 

3n A 1) "d) 

*S;Yi|( 7 (Si(n,n);j/i|Ui) > logTV + 7 , 


( 68 ) 


Then we can bound 


P[C(C/i, 5 *,T*) >r;] 

= P[ f| {(C/i, 5 i(l,n),ri(l, 0 )^ J-„}] 

l<n<Af.l<f<L 

(78) 


<F[{U,S,T)^TyUS] 
min{TV, L} - 1 
7VL(exp(— 7 ) — exp(— 27 )) 


(79) 


Denote by S* and T* the coefficients selected by the 
encoder and Yj*, Y 2 the corresponding outputs. Averaged 
over the codebook, we can bound 


where we invoked Lemma 5 in (79), and defined the set 

S := {iu,s,t) : is-,T\u{s;t\u) > logNL - 27 }. (80) 


P[f2] 

= P[ y {ius-,YiiUmSmid,d);Y-^) > logMN+ j}] 

(69) 

< {M - 1)MF[ius;yAU 2S2{1,1)]Y*) > \ogMN + -f\YA 

(70) 

M- 1 


< 


M exp( 7 ) 


(71) 


Combining (76) and (79) we have 

P[^lU^3] 

< [ P[(C/,S',r) ^ J-„U5]du 


/o 


I _ min{jV,L}-l I ^-exp(7) ( 81 ) 

iVL(exp(— 7 ) — exp(— 27 )) 

but by definitions (60) and (77), 


by change of measure, where the joint probability of 

{Um, Sm{d,d)}^^^ j and Y* satisfies Pu^Sm(d4)Y‘ ~ 
Pus X Py{- Similarly 

P[f4] 

= 1P[ U {*S;Yi|c/(>S'i(n,n); Yi*|C/i) > log7V + 7}] 


n^l,l<n<N 



(72) 

< (TV - l)TVEP[rs.y7c(^i(2,1); Y^Ui) 


>\ogN+ -f\Y*U4i 

(73) 

^ (TV-l)TV 
~ TV exp ( 7 ) 

(74) 

where Psi(n.n)Ypc/i = PuPs\uPy*\u for n 7 ^ 1 . 

notice that 

Next, 

Pi^iUfs] <EC(C/i,S'*,T*) 

(75) 

= fF\aUuS\T*)>v]dv. 

Jo 

(76) 



P[(C/,S',r) i U5]du 


= / Ei{([/,S',r) € 

Jo 

= E[ 1 {(C/, T)eS} + C1{(C/, T) i 5}] 
= P[(C/,5,T)e5] 


+ J dPuSTPYiY2\USTiyUST)^{{U, S,T) ^ S} 


(82) 

(83) 


(84) 


= P[(C/, T)g 5] 

+ / dPy^Y 2 ( 7 STl{(Pl, Y 2 ) G yuST U (U, S,T) ^ 5} 

(85) 

= F[iU, 5, T) G 5 U (Yi, ^ 2 ) G yusr] ( 86 ) 

= P[{zs;r|c/(^; T\U) > log NL - 27 } 

'J bus-,Yi{US-,Yi) < logMlV + 7 } 

U {*c/T;Y 2 {UY ; Y 2 ) < log ML + 7 } 

^ bs-,Yi\u{S;Yi\U) < loglV + 7 } 
U{tT;Y 2 |c/(T;Y 2 |C/) <logL + 7 }]. (87) 


Note that {S*,T*) has a complicated distribution since 
the coefficients are selected through the minimization of 
(59). To tackle this, let 


Finally, the proof is accomplished by substituting (87) 
into (81) and then applying the union bound with (71), 
(74) and (81). 


Tv := {{u, s,t) & X S X T : s, t) < w}. (77) 


Remark 11. In the i.i.d. setting Theorem 10 readily gives the 



following achievable region 
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Rq 


Ri 

Rii 

Rii 


Rii 

-4 

Ri 
h Ri 


<R^ 

>/(Ui;U 2 |Uo) 

</(UoU,;Y,) 

</(U.;Y,lUo) 


( 88 ) 


for some Ra^Ri > 0, i = 1 , 2 , PuoUiU 2 function x : 
Uo xUi XU 2 X. Fourier-Motzkin elimination gives the 
same region as in [ 6 ]; see also [1, Theorem 8.4]. 


V. Discussion 

In contrast to the one-shot mutual covering lemma, a one- 
shot version of mutual packing lemma [1, Lemma 12.2] holds 
trivially, because the key step union bound in the proof of [5, 
Lemma 2] does not require independence among the pairs. 

Lemma 12. Fix {Pxy , M, N,^), then 

P[maxix;y(2fm; L„) > log MiV -f 7 ] < exp(- 7 ) (89) 

m,n 

where for all 1 < m < M, \ < n < N, PxmYn = Rx x Py- 

This can be used to derive a one-shot version of Berger- 
Tung inner bound without time sharing or stochastic decoders. 
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